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(54) Method and apparatus for image sensing v«rith dynamic range expansion 

(57) An image sensing apparatus which senses an 
object a plurality of times in different exposures to obtain 
a plurality of images and makes a single image with a 
widened dynamic range by processing the plurality of 
sensed images, generates a histogram of signal levels 
of an image sensed in a proper exposure by using a his- 
togram generator, automatically decides whether a 
dynamic range expansion processing is to be proc- 
essed or not on the basis of the generated histogram by 
using a dynamic range expansion deciding unit, deter- 
mines a parameter for sensing another image of the 
object in an exposure which is different from the proper 
exposure in accordance with a decision by the dynamic 
range expansion deciding unit by using a parameter 
determination unit, and obtains a single image of the 
object l3y processing the image sensed in the proper 
exposure and the image sensed by using the parameter 
determined by said parameter determination unit an 
image processing unit. When an image sensing appara- 
tus is not equipped with a photometer, a pre-sensing 
operation is performed, an image sensed in the pre- 
sensing operation is analyzed to find a proper exposure, 
and this image is also used in the dynamic range expan- 
sion processing. A part or whole image data of a plural- 
ity of images which are obtained as above and 
necessary for the dynamic range expansion processing 
are compressed by a compressing unit and the com- 
pressed image data is used in the dynamic range 
expansion processing. 
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Description 

The present invention relates to an image sensing 
apparatus and method and. more particularly, to an 
image sensing apparatus and method capable of 
speeding up and effectively performing an operation for 
obtaining a single image with a wide dynamic range by 
processing a plurality of images obtained by sensing an 
identical scene a plurality of times in different expo- 
sures. 

Some image sensing apparatuses, such as a video 
camera, a still video camera, and an electrophoto- 
graphic camera, use a solid-state electric image sensor, 
such as a CCD. which generates image signals repre- 
senting a sensed irnage. The dynamic range of the 
solid-state electric image sensing device for incoming 
light is much narrower than that of silver halide film and 
the like. Therefore, when a scene to be sensed includes 
a bright part and a dark part and luminosity difference 
between them is large, it is very difficult to sense kx>th 
part in an exposure proper for the solid-state electric 
image sensing device in a single image sensing opera- 
tion. In most cases, if the exposure for sensing the 
scene is adjusted to the dark part of the scene, then the 
bright part may be expressed in white, and if the expo- 
sure is adjusted to the bright part, then the dark part 
may be expressed in blacK for example. 

In order to overcome the aforesaid problems, as a 
method of expanding the dynamic range of a solid-state 
image sensing device, a technique is proposed in e.g., 
Japanese Patent Laid-Open No. 63-306778. in which 
an identical scene is sensed in different exposures to 
obtain image data of a plurality of sensed images, and 
the image data of the sensed images is processed in 
accordance with some predetermined operation to 
obtain a single image with an expanded dynamic range. 
Further. Japanese Patent Laid-Open No. 5-64075. titled 
as "Video Camera. Image Sensing Method Using the 
Video Camera, and r^ethod of Operating the Video 
Camera, and Image Sensing Apparatus and Method", 
proposes a camera and a method for expanding a 
dynamic range, which senses an identical scene in an 
exposure that is proper to a dark area of the scene to 
obtain the first image and in an exposure that is proper 
to a bright area to obtain the second image, then 
replaces the image data of the bright area in the first 
image by the corresponding area in the second image 
so as to ofcjtain a composite image, and obtains a single 
image including the dark and bright areas both sensed 
in the proper exposures. 

However, in the aforesaid camera capable of 
expanding its dynamic range, there were following prob- 
lems, namely, 

L Two images, i.e.. the first image obtained when an 
exposure is adjusted to the dark part of the scene to 
be sensed and the second image obtained when an 
exposure is adjusted to the bright part of the scene, 
are always sensed. Therefore, when a user does 



not want dynamic range expansion, another image 
has to be sensed with an exposure adjusted in 
accordance with the brightness of the entire scene 
(the image sensed as this is called "standard 

5 image^ and the exposure adjusted as this is called 
as "proper exposure", hereinafter), which is trouble- 
some and operation time for sensing an image is 
long, and a user may miss a shutter chance. 
II. If an electrophotographic camera does not have 

10 a photometer, the camera has to perform an image 
sensing operation (called as "pre-sensing opera- 
tion", hereinafter) just for deciding an exposure for 
the subsequent image sensing operation. Then, 
after a proper exposure for obtaining a standard 

IS image is determined on the basis of image data 
obtained in the pre-sensing operation and the used 
exposure, the second image sensing operation is 
performed in the proper exposure (called as "main 
sensing operation", hereinafter). In the above- 

so described image sensing operation, the image data 
obtained in the pre-sensing operation is used just 
for determining an exposure to be used in the sub- 
sequent image sensing operation, and abandoned 
thereafter 

25 In addition, in order to process a plurality of 

images to obtain a single image with the expanded 
dynamic range in accordance with the image sens- 
ing processing described in the previous para- 
graph, in order to obtain the plurality of images, it is 

30 necessary to determine the proper exposure by 
performing the presensing operation, then to per- 
form main image sensing operation in the proper, 
over, and under exposure. 

However, as described above, image data 

35 obtained in the pre-sensing operation is conven- 
tionally used only for photometry, and abandoned 
thereafter. Consequently, it is necessary to perform 
image sensing operation at least three times for 
obtaining a plurality of images (at least two images) 

40 in different exposures, which is troublesome and 
consumes a long time. 

Furthermore, since an image is sensed in the 
pre-sensing operation, and more than two images 
are sensed in the main sensing operation, there is 

45 a time gap between when an image is sensed for 
the first time and when an image is sensed in the n- 
th time. Accordingly, a problem arises that images 
to be processed may be considerably different from 
each other. 

50 III. Since it is necessary to store a plurality of 
sensed images to perform the aforesaid dynamic 
range expansion processing, there is a problem 
that an amount of image data to be stored is quite 
large. 

55 

The present invention has been made in considera- 
tion of the above situation, and has as its object to pro- 
vide an image sensing apparatus capable of 
automatically determining whether it is necessary to 
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perform dynamic range expansion processing or not as 
well as capable of perfbrnntng image sensing operation 
effectively. 

In a first aspect, the present invention provides an 
image sensing apparatus which senses an object a plu- 
rality of times in different exposures to obtain a plurality 
of images and makes a single image with a widened 
dynamic range by processing the plurality of sensed 
images, the apparatus comprising: histogram generat- 
ing means for generating a histogram of signal levels of 
a first image sensed in a first exposure; deciding means 
for deciding whether a dynamic range expansion 
processing is to be processed or not on the basis of the 
histogram generated by the histogram generating 
means; parameter determination means for determining 
a parameter for sensing a second image of the object in 
a second exposure which is different from the first expo- 
sure in accordance with a decision by the deciding 
means; and image processing means for obtaining a 
single image of the object by processing the first image 
and the second image which is sensed by using the 
parameter determined by the parameter determination 
means. 

With the above configuration, when it is determined 
that it is not necessary to perform the dynamic range 
expansion operation, a single scene does not have to 
be sensed a plurality of times unnecessarily, thereby 
making the image sensing operation effective. 

It is another object of the present invention to pro- 
vide an image sensing apparatus and method capable 
of shortening time for image sensing operation for 
obtaining a single image with expanded dynamic range 
from a plurality of images sensed in different exposures 
by using an image sensed for photometry to determine 
an exposure to be used in the main image sensing oper- 
ation and to perfbmi the dynamic range expansion 
processing, as well as capable of making a photometer 
unnecessary. 

in a second aspect, the present invention provides 
an image sensing apparatus which senses an object a 
plurality of times in different exposures to obtain a plu- 
rality of images and makes a single image with a wid- 
ened dynamic range by processing the plurality of 
sensed images, the apparatus comprising: an image 
sensor for sensing the object so as to obtain an image 
and outputting the obtained image; opa-ation means for 
finding a proper exposure by using a first image sensed 
in a first image sensing operation and a first exposure 
used to sense the first image; deciding means for decid- 
ing whether a dynamic range expansion processing is 
to be performed or not; image sensor control means for 
controlling the image sensor in order to sense a second 
image in an exposure which is different from the first 
exposure, on the basis of the proper exposure ol>tained 
by the operation means and the first exposure; and 
image processing means for, depending upon a deci- 
sion by the deciding means, generating a single image 
by using the first Image and the second image. 



With the aforesaid configuration, an image sensed 
for photometry is also used in the dynamic range expan- 
sion processing, thereby obtaining the necessary 
number of images for dynamic range expansion 

5 processing in the less number of image sensing opera- 
tions, as well as making a photometer unnecessary. 

It is another ot>ject of the present invention to pro- 
vide an image sensing apparatus capable of reducing 
an amount of data to be stored which is used in the 

10 dynamic range expansion processing. 

A third aspect of the present invention provides ah 
image sensing apparatus which senses an object a plu- 
rality of times in different exposures to obtain a plurality 
of images and makes a single image with a widened 

15 dynamic range by processing tiie plurality of sensed 
images, tiie apparatus comprising: first compressing 
means for generating compressed data by compressing 
image data of an image which is other than a first image 
that is one of the plurality of images; and storage means 

20 for storing image data of the first image and the com- 
pressed data of the image which is other than the first 
image. 

Such an image sensing apparatus may further 
comprise: area determination means for determining an 

25 area which needs to be performed with a dynamic range 
expansion processing in the first image; and extraction 
means for extracting an area corresponding to the area 
determined by the area determination means in the 
image which is other than the first image, wherein the 

30 first compressing means compresses image data of the 
area exti'acted by the extraction means. 

Other features and advantages of the present 
invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, 

35 in which like reference characters designate the same 
or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

40 The accorrpanying drawings, which are incorpo- 
rated in and constitute a p>art of the specification, illus- 
trate emtxxjiments of the invention and. together with 
the description, serve to explain the principles of the 
invention. 

45 

Fig. 1 is a block diagram illustrating a configuration 
of an image sensing apparatus according to a first 
embodiment of the present invention; 
Rg. 2 is a histogram showing signal levels of a 

so standard image and the number of events; 

Fig, 3 is another histogram showing signal levels of 
the standard image and the number of events; 
Rg. 4 is a block diagram illustrating a configuration 
of an image processing unit according to the first 

55 embodiment and its modification; 

Rg. 5 is a block diagram illustrating an Image sens- 
ing apparatus and an image processing apparatus 
according to the modification of the first embodi- 
ment of the present invention; 
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Fig. 6 is a block diagram illustrating a configuration 
of an image sensing apparatus according to a sec- 
ond embodiment and its modification; 
Rgs. 7A and 7B are explanatory charts for explain- 
ing sequences of controlling a solid-state image 
sensor according to the second embodiment; 
Rgs. 8A to 8D are explanatory charts for explaining 
sequences of controlling the solid-state image sen- 
sor according to the modification of the second 
embodiment; 

Rg. 9 is a fk)wchart showing an operation accord- 
ing to the modification of the second en^X)diment: 
Rg. 10 is a block diagram illustrating a configuration 
of an image sensing apparatus according to a third 
embodiment; 

Rgs. 11 A and 11 B are two images obtained by 
sensing an identical scene in different exposures, 
especially, Rg. 11 A is the image sensed by adjust* 
ing the exposure so as to be proper to inside of an 
room, and Rg. 11 B is the image sensed by lowering 
the exposure so that a part of the solid-state image 
sensor corresponding to an area which is outside of 
the room is not saturated; 

Rg. 12 is a graph showing luminance level along 
with a horizontal line of the image shown in Rg. 
11 A; 

Rg. 13 Is an example of dividing an image used in 
dynamic range expansion processing into predeter- 
mined sized areas each of which is a unit subjected 
for determining whether or not the area is needed to 
be performed with dynamic range expansion 
processing; 

Rg. 1 4 Is a block diagram illustrating a configuration 
of an image sensing apparatus according to a first 
modification of the third embodiment; and 
Rg. 1 5 is a block diagram illustrating a configuration 
of an image sensing apparatus according to a sec- 
ond modification of the third embodiment. 



DFTAILED DES CRIPTION OF THE PREFERRED 
EMBODIMENTS 



Preferred embodiments of the present invention will 
be described in detail in accordance with the accompa- 
nying drawings. 

(First Embodiment) 

A first embodiment of the present invention is 
described below. Rg. 1 is a block diagram illustrating a 
configuration of an image sensing apparatus according 
to a first embodiment of the present invention. Referring 
to Fig. 1 . f k>w of image data and image sensing opera- 
tion will be described. 

An image of a scene passes through a lens 1. an 
optical low-pass filter 2, and an iris diaphragm 3, and is 
formed on photo sensing surface of CCD 4. The iris dia- 
phragm 3 and an exposure period of the CCD 4 (elec- 
tronic shutter speed) are controlled by an exposure 



controller 1 1 on the basis of data from a photometer (not 
shown) so that the CCD 4 is properly exposed. Analog 
image data of the first image (standard image) formed 
on the CCD 4 by sensing the scene in the proper expo- 

5 sure (first exposure) , is converted into digital image data 
by an analog-digital (A/D) converter 5. and stored in a 
memory 7. The digital image data stored in the memory 
7 is sent to a histogram generator 8 where a histogram 
of the number of event of signal levels of the standard 

10 image data, as shown in Rg. 2, is generated. 

Thereafter, a dynamic range expansion deciding 
units (referred as "deciding units", hereinafter) decides 
whether a dynamic range expansion processing is to be 
performed or not on the basis of the generated hislo- 

15 gram. This decision is made in accordance with the total 
numbers of events of signal levels of the standard image 
data in different ranges A and B which are in high-end 
and low-end regions of the dynamic range of the image 
sensing apparatus, and predetermined threshokJs of 

20 the numbers of events. TA and TB. For example, let the 
central values of the ranges A and B, LA and LB, 
respectively, have relationship LA < LB. and let the total 
numbers of the events of signal levels in the ranges A 
and B be HA and HB, respectively Then, if the following 

2$ condition 1 is satisfied, it is decided that the dynamic 
range expansion processing is not performed. 

KIA < TA and HB < TB Condition 1 

30 If the deciding unit 9 decided not to perform the 

dynamic range expansion processing, then the stand- 
ard image data stored in the memory 7 is read and out- 
putted through the image processing unit 12 without 
changing the standard image data. In contrast, if the 

35 deciding unit 9 decided to perform the dynamic range 
expansion processing, then the parameter determina- 
tion unit 10 determines parameters, i.e., a value for the 
iris diaphragm and an electronic shutter speed, for 
sensing the second image as a non-standard image in 

40 the second exposure on the basis of the histogram gen- 
erated by the histogram generator 8. 

For the simplicity sake, the value for the iris dia- 
phragm is kept unchanged since the standard image is 
sensed, and only the electronic shutter speed is altered 

45 in order to obtain the non-standard image. For example, 
referring to Fig. 2, conditions for performing the dynamic 
range expansion processing can be divided as follow. 



so 



55 



HA < TA and HB > TB Condition 2 

HA > TA and HB < TB Condition 3 

HA > TA and HB > TB Condition 4 

HA < TB and HA < TB Condition 5 

If the condition 2 is satisfied, for instance, it is 
known that there is a relatively large bright area in the 
standard image. Accordingly, the electronic shutter 
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speed for sensing a non-standard image (the second 
image) is determined so that an exposure period in the 
second image sensing operation is shorter than the 
exposure period when the standard Image is sensed. 
Similarly, if the condition 3 is satisfied, the electronic 
shutter speed is determined so that the exposure period 
tor sensing the non-standard image is longer than the 
exposure period for sensing the standard image so as 
to adjust the exposure period to the dark part of the 
standard Image. The following table 1 shows ratios of 
the electronic shutter speed, used for sensing a non- 
standard image, with respect to the shutter speed used 
for sensing the standard image in the first emt>odiment. 
Note that the ratios of the electronic shutter speed, R. 
are decided in accordance with a ratio of the total num- 
bers of events of signal levels within the ranges A and B, 
denoted by HA and HB. respectively, with respect to the 
threshold values. TA and TB, and SA HAHA) and SB 
(= HB/TB) 



Table 1 



Electronic Shutter Speed Determination Table 


S 


SA(=HAAA) 


SBlirHBH'B) 


1 ^S<2 


x2 


x1/2 


2^S<3 


x4 


X 1/4 


S^3 


x8 


x1/8 



The electronic shutter speed is determined on the 
basis of the larger value of SA and SB by referring to the 
table 1. For example, in a case where the condition 4 is 
satisfied and 1 ^ SA < 2 and 2 ^ SB < 3. since SB is 
larger than SA, the electronic shutter speed is deter- 
mined on the basis of SB. From the table 1 . the elec- 
tronic shutter speed for sensing a non-standard image 
is determined as 1/4 of the shutter speed when the 
standard image Is sensed. 

in the method as described above, a non-standard 
image is sensed and processed with the standard 
image. In order to sense a plurality of non-standard 
images, a plurality of electronic shutter speed can be 
determined on the basis of both SA and SB. For exam- 
ple, ill s SA < 2 and 2 ^ SB < 3, then the first electrical 
shutter speed for sensing the first non-standard image 
cr... be decided on the basis of SA and is twice as fast 
as the shutter speed used for sensing the standard 
image, and the second electrical shutter speed for sens- 
ing the second non-standard image can be determined 
on the basis of SB and is 1/4 of the shutter speed used 
for sensing the standard image. 

The value for the iris diaphragm and the determined 
shutter speed for sensing a non-standard image is sent 
to the exposure controller 1 1 which controls the iris dia- 
phragm 3 and the CCD 4. and a non-standard image is 
obtained. Next, the image data of the non-standard 



8 

image is analog-digital converted, and the converted 
image data is stored in a memory 6. The image data 
stored in the memory 6 and the image data stored in the 
memory 7 are sent to the image processing unit 12 

5 where these two sets of data are processed and made 
into a single image with a widened dynamic range. 

Rg. 4 is a block diagram illustrating a configuration 
of the image processing unit 12 according to the first 
embodiment of the present invention. 

TO In Fig. 4, reference numeral 51 denotes a lumi- 
nance level adjusting unit for changing the luminance 
level of the non-standard image to match to the lumi- 
nance level of the standard image; 52. a replacing area 
determination unit for determining an area to be 

15 replaced in the standard image by a corresponding area 
of the level-adjusted non-standard image; and 53, a 
replacing unit for replacing the determined area of the 
standard image by the corresponding area of the non- 
standard image. 

20 In the first embodiment, the image sensing appara- 
tus decides whether the dynamic range expansion 
processing is to be performed or not on the basis of the 
histogram of the signal levels of the standard image, 
however, a mode with which a user can make the afbre- 

25 said decision may be also provided in the image sensr 
ing apparatus. 

Further, the electronic shutter speed is decided on 
the basis of the two ranges A and 6 in the histogram of 
the signal levels as shown in Fig. 2, however, the 

30 present invention is not limited to this. For example, in 
order to improve analyzing quality of the standard 
image, the shutter speed can be decided on the basis of 
more than two different ranges, A to D. as shown in Fig. 
3. in the histogram of the signal levels. Furthermore, a 

35 mode for which a user can set a value for the iris dia- 
phragm and a shutter speed for sensing a standard 
image and a non-standard image can be provided in the 
image sensing apparatus. It is also possible to change 
the values of the iris diaphragm by the parameter deter- 

40 mination unit 1 0 for sensing a non-standard image. 

According to the image sensing apparatus of the 
first embodiment as described above, a starxjard image 
is sensed first, then it is determined whether the 
dynamic range expansion processing is to be per- 

45 formed or not on the basis of a histogram generated in 
accordance with the data of the sensed image, and a 
parameter to be used for sensing a second Image is 
determined when the dynamic range expansion 
processing is determined to be processed. In other 

so words, the image sensing apparatus automatically 
decides whether the dynamic range of the first image 
needs to be expanded or not, and. if H does, the image 
sensing apparatus senses the second image in an 
exposure which is adjusted to a bright area or a dark 

55 area, or senses two second images In exposures 
respectively adjusted to bright and dark areas. Thereaf- 
ter, the image sensing apparatus processes the first 
image and the second image or images, thereby obtain- 
ing a single image with a widened dynamic range when 
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the difference between luminance levels in bright and 
dark parts of a scene is very large. Further, it is deter- 
mined that the dynamic range of the first image needs 
not to be expanded, the first image is directly outputted. 
thereby shortening time for an image sensing operation. 

(Modification of the First Embodiment) 

A modification of the first embodiment will be 
described below. As shown in Fig. 5, the dynamic range 
expansion processing is performed by using a host 
computer 49 capable of controlling an image sensing 
apparatus 41 . First, image data of an image sensed with 
a proper exposure (standard image) sensed by the 
image sensing apparatus 41 is transmitted to the host 
computer 49. Then, the image data of the starxiard 
image is stored in a memory 44, and a histogram based 
on the stored image data is generated by a histogram 
generator 45, similarly to the first embodiment. Next, the 
generated histogram is sent to a dynamic range expan- 
sion deciding unit 46 (referred as "deciding unit 46". 
hereinafter) which decides whether the dynamic range 
expansion processing Is to be performed or not in 
accordance with the histogram. 

If the deciding unit 46 decided that it is not neces- 
sary to perform the dynamic range expansion process- 
ing, then the standard image is unprocessed and 
outputted through an image processing unit 48. In con- 
trast, the deciding unit 46 decided that it is necessary to 
perform the dynamic range expansion processing, a 
parameter determination unit 47 determines a value for 
the iris diaphragm and an electrical shutter speed for 
sensing a non-standard image in accordance with the 
generated histogram, as in the first embodiment. The 
determined value for the iris diaphragm and the electri- 
cal shutter speed are transmitted via an interface 42 to 
the image sensing apparatus 41 which senses a non- 
standard image by using the transmitted parameters. 

Image data of the sensed non-standard image is 
sent the host computer 49 and stored in a memory 43. 
The image data of the standard image stored in the 
memory 44 and the image data of the non-standard 
image stored in the memory 43 are processed in the 
image processing unit 48. and made into a single image 
with a widened dynamic range. The obtained image 
may be outputted, or may be sent to the image sensing 
apparatus 41 and stored in a memory or a storage 
medium provided in the image sensing apparatus, in 
accordance with a user's choice. The image processing 
unit 48 has a configuration shown in Rg. 4, for example, 
as the image processing unit 1 2 in the first embodiment. 

According to the modification of the first embodi- 
ment, an image with a widened dynamic range can be 
obtained, as in the first embodiment, by using an image 
sensing apparatus which does not have a dynamic 
range expansion function by connecting it to a host 
computer Ceipable of performing the dynamic range 
expansion processing by processing images transmit- 
ted from the image sensing apparatus. 



It should be noted that, in the first embodiment and 
its modification, the image sensing apparatus has a 
photometer, however, it is possible to apply the afore- 
said configuration to an image sensing apparatus which 

5 does not have a photometer. In such a case, an image 
to be analyzed for photometry is sensed f irst (pre-sens- 
ing operation), then by controlling an iris diaphragm and 
an electronic shutter on the basis of the result of the pre- 
sensing operation, a standard image can be obtained. 

10 The decision whether the dynamic range expansion 
processing is to be performed or not can be made by 
performing the above operation on the obtained stand- 
ard image. 

75 <Seoond Embodiment) 

Fig. 6 is a block diagram illustrating a configuration 
of an image sensing apparatus according to a second 
embodiment. In Fig. 6, reference numeral 61 denotes a 

20 lens; 62, an optical low-pass filter; 63, an image sensor 
such as CCD; 64, an analog-digital (A/D) converter; 65, 
an image processing unit; 66. a memory; and 67, an 
image sensor controller Figs. 7A and 7B are explana- 
tory charts for explaining sequences of controlling an 

25 image sensor 63 and exposures in the second embodi- 
ment. 

An operation of the second emlxxJiment will be 
described below with reference to Figs. 6. 7A and 7B. 
First, a pre-sensing operation for sensing a first image 

30 used for photometry is performed, in the pre-sensing 
operation, referring to Fig. 7A. signals in two interlaced 
fields of the image sensor 63 are outputted. One of the 
aforesaid two interlaced fields is called A field, and the 
other is called B field, hereinafter. The image data for 

35 photometry outputted from the image sensor 63 is con- 
verted into digital image data by the A/D converter 64, 
and inputted to the image processing unit 65 together 
with information on the exposure used fa sensing the 
first image (first exposure). The image processing unit 

40 65 decides an exposure to be used in tiie next image 
sensing operation by performing calculations based on 
the inputted digital image data and the first exposure, 
and serKjs a signal indicating the decided exposure to 
the image sensor confroller 67. The Image sensor con- 

45 troller 67 can control tiie image sensor 63 in the next 
image sensing operation in accordance with the 
decided exposure. The image data of the first image is 
stored in the memory 66. 

Next, a main sensing operation is performed. As 

so shown in Fig. 7A, if the pre-sensing operation is per- 
formed in an improper exposure, e.g., the over exposure 
(71) or in the under exposure (72). ttie image sensor 
controller 67 controls the image sensor 63 so that the 
both images in the A and B fields are to be sensed in ttie 

55 proper exposure in the main sensing operation. In a 
case where a pre-sensing operation is performed in the 
proper exposure, the luminance distritjution of the entire 
first image data obtained in the pre-sensing operation is 
analyzed. As the analyzed result, if the first image is 
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found that it contains a relatively large dark area (73). 
images in the A and B fields are sensed in an over expo- 
sure in the main sensing operation. 

In contrast, if the first image is found that it contains 
a relatively large bright area (74). then images in the A 
and B fields are sensed in an under exposure in the 
main sensing operation. Further, if the luminance distri- 
bution of the first image shows a proper distrit^ution 
(75). then an image in the A field is sensed in an over 
exposure and an image in the B field is sensed in an 
under exposure. The image data of the A and B fields, 
outputted from the image sensor 63 is stored in the 
memory 66 through the image processing unit 65. simi- 
larly to the first image sensed in the pre-sensing opera- 
tion. 

In a case where the differences between the expo- 
sure used in the pre-sensrng operation and the proper 
exposure is larger than a predetermined value, the 
dynamic range expansion processing is performed with- 
out the image data obtained in the pre-sensing opera- 
tion. Of course, it is possible to perform the dynamic 
range expansion processing by using only the image 
data obtained in the main sensing operation. Further, in 
a case where a user does not want the dynamic range 
expansion, the dynamic range expansion processing is 
not performed, and image data of a single image is 
made out of image data of two fields which are sensed 
in t>oth the pre-sensing operation and the main sensing 
operation in the identical and proper exposure. The 
image data is converted into luminance signals and 
color difference signals and outputted. 

Further, when the dynamic range expansion 
processing is to be performed, first, a plurality of images 
are sensed in different exposures, then by processing 
image data of these sensed images, a single image with 
a widened dynamic range is made and outputted. As a 
dynamic range expansion processing method, methods 
disclosed in Japanese Patent Laid Open Nos. 7-135569 
and 7-131799 or Japanese Patent Application Nos. 6- 
285067 and 6-285066 (USP 08/559.074). all of which 
are applied by the present applicant, can be used. 

Fig. 78 shows an example when only image data in 
the A field of the image sensor 63 is outputted in the 
pre-sensing operation so as to shorten time consumed 
in the overall image sensing operation. Referring to Fig. 
7B. if the pre-sensing operation of the A field is per- 
formed In an improper exposure (76 or 77), kx>th the A 
and B fields are sensed in the proper exposure in the 
main s«* ^sing operation. Whereas, if the pre-sensing 
operation is performed in the proper exposure (78 ~ 
80), in the main sensing operation, the Afield is sensed 
in the proper exposure and the B field is sensed in either 
the proper, over or under exposure on the basis of the 
luminance distribution of the first image. Image data of 
the images sensed as above is transmitted to the image 
processing unit 65. 

In a case where it is not necessary to perform the 
dynamic range expansion processing, a single image is 
made from image data of two fields sensed in the iden- 



tical and proper exposure out of image data of the three 
fields sensed as above in different exposures. Further, 
in a case where the dynamic range expansion process- 
ing is to be performecftk a single image with a widened 

5 dynamic range is made by processing the image data of 
the three fields. 

As described above, by processing image data 
sensed in a proper or improper exposure together with 
image data for photometry, it is possible to obtain image 

10 data of a plurality of images for the dynamic range 
expansion processing by performing a pre-sensing 
operation and only a single main sensing operation. 
Accordingly, the image data for photometry sensed in 
the pre-sensing operation is not wasted, and an image 

75 with a widened dynsunic range can be obtained in a 
short time. 

Further, by sensing image data of a single field in 
the pre-sensing operation as expressed in Fig. 7B, time 
for the overall image sensing operation can be further 
20 shortened. 

{Modification of the Second Enrti3odiment> 

Rgs. 8A to 8D are explanatory charts for explaining 
25 sequerrces of controlling the image sensor 63 according 
to a modification of the second embodiment. Note that 
an image sensing apparatus used in this modification is 
the same as the one shown in Fig. 6. The feature of the 
image sensing apparatus of the modification is that 
30 image data of three kinds of images, respectively 
ot}tained by sensing in proper, over and under expo- 
sures, is used in the dynamic range expansion process- 
ing so as to cover a wide luminance range spread from 
a bright area to a dark area of a scene. Further. Fig. 9 is 
35 a flowchart showing an operation of the image sensing 
apparatus according to the modification of the second 
embodiment. 

Refening to Figs. 8A to 8D and Fig. 9. an operation 
for obtaining data from the image sensor 63 will be 

40 described. First at step S1. a pre-sensing operation is 
performed, then at step S2, an image obtained in the 
pre-sensing operation (first image) is stored In the mem- 
ory 66, and at step S3, a proper exposure is deter- 
mined. Next at step S4, whether the dynamic range 

45 expansion processing is to be performed or not is 
decided. When it is decided that the dynamic range 
expansion processing does not need to be performed, 
the A and B fields are sensed in an exposure shown in 
Fig. 80 in the main sensing operation at step S5. 

so Whereas, at step S4. if it is determined that the 
dynamic range expansion processing is to be per- 
formed, the exposure used in the pre-sensing operation 
(first exposure) is compared to the proper exposure at 
step S6. and if the difference between the two expo- 

55 sures is under a predetermined value, the scene is 
sensed in an exposure shown in Fig. 8A at step S7 or in 
an exposure shown in Fig. 8B at step S8 in the main 
sensing operation. Referring to Fig. 8A. image data of a 
single frame (consists of the A and B fields) is outputted 
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from the image sensor 63 for photometry in the pre- 
sensing operation, thereafter, image data of two frames 
is outputted in the subsequent main sensing operations. 
If the first image is sensed in an over exposure (81), 
image data of two frames are sensed in the proper and 
under exposures in the main sensing operations. Then, 
the image data of the three frames obtained as above is 
processed as in the second embodiment to expand a 
dynamic range of the image sensing apparatus. Fur- 
ther, if the first image is sensed in an under exposure 
(82). image data of two frames are sensed in the proper 
and over exposures. Furthermore, if the first image is 
sensed in the proper exposure (83), image data of two 
frames are sensed in an over and under exposures. 

Fig. 8B shows an example to shorten time for the 
overall image sensing operation by sensing an image of 
either the A or B field in the pre-sensing operation, and 
image data of three fields, including image data of 
improper exposure, are obtained in the main sensing 
operation. In this method, time to perform the overall 
image sensing operation is shorter by time tor obtaining 
image data of two fields, comparing to the image sens- 
ing operation according to Hg. 8A. First, in the pre- 
sensing operation, image data of the A (or B) field is 
obtained. If the pre-sensing operation was performed in 
an over exposure (84), image data of both the A and B 
fields is obtained in the proper exposure, then image 
data of the B (or A) field is obtained in an under expo- 
sure. In a case where the pre-sensing operation was 
performed in an under exposure (85). image data of 
both the A and B fields is obtained in the proper expo* 
sure, then image data of the B (or A) field is obtained in 
an over exposure. If the pre-sensing operation was per- 
formed in the proper exposure (86), the next field is 
sensed in an over exposure, in the proper, then in an 
under exposures. 

Further, the first exposure is compared to the 
proper exposure, and if it is determined at step 86 that 
the difference between the exposures is over a prede- 
termined value, since it is not proper to use the first 
image data in the dynamic range expansion processing 
(the first image Is too bright or too dark), the image data 
stored in the memory 66 is deleted at step S9. Thereaf- 
ter, main sensing operations are performed in the expo- 
sures as shown in Fig. 8D at step S10. More specifically, 
in addition to a standard image (i.e.. image sensed in 
the proper exposure), image data of at least one frame 
is newly sensed in an improper exposure, then by 
processing the image data of a plurality of frames, a sin- 
gle image with a widened dynamic range is obtained. 

By sensing a scene in the method as described 
above in accordance with exposures set in the aforesaid 
sequences, image data for photometry sensed in a pre- 
sensing operation Is not wasted, a plurality of images 
necessary for the dynamic range expansion processing 
can be sensed in different exposures by performing the 
main sensing operations one time less than the conven- 
tional image sensing operation, and a single image with 



a widened dynamic range can be obtained by process- 
ing the plurality of sensed images. 

According to the second embodiment and its modi- 
fication as described above, image data for photometry 

5 (first image) obtained in a pre-sensing operation and 
image data sensed in the proper or improper exposure 
decided in accordance with the first image is used in the 
dynamic range expansion processing. Therefore, the 
image data obtained in the pre-sensing operation is not 

10 wasted as well as overall image sensing time can be 
shortened. Especially, in the dynamic range expansion 
processing performed on the basis of two images 
sensed in the proper exposure and in the improper 
exposure, it becomes unnecessary to sense a third 

15 image in the main sensing operation. 

Further, by obtaining image data of only one field 
for photometry in a pre-sensing operation and process- 
ing the obtained image data with another image data of 
a frame obtained in the main sensing operation, it is 

20 possible to further shorten the overall image sensing 
time. 

(Third Embodiment 

25 A third embodiment is described below. 

Fig. 10 is a block diagram illustrating a configuration 
of an image sensing apparatus according to the third 
embodiment, in Fig. 10, reference numeral 101 denotes 
a lens; 102, an iris diaphragm for adjusting an amount of 

30 incoming light; 103. an optical low-pass fiter; and 104, 
a solid-state image sensor such as CCD. 

Further, reference numeral 105 denotes an expo- 
sure controller for controlling an exposure decided by 
the value of the iris diaphragm 1 02 and shutter speed of 

35 the solid-state image sensor 104. Reference numeral 
1 06 denotes an A/D converter for converting image data 
outputted from the solid-state image sensor 104 to dig- 
ital image data; and 107 and 108 are frame memories 
for temporarily storing digital image data of a single 

40 frame. 

Reference numeral 109 denotes a replacing area 
determination unit for determining areas which is to be 
processed with dynamic range expansion processing 
(i.e.. bright or dark areas in the image); 1 10. a compres- 
45 sion unit for compressing either image data of a refer- 
ence image or of an image other than the reference 
image (non-reference image) which will be explained 
later, stored in the frame memory 1 07 or 1 08; and 111. 
an storage unit for storing the image data of the refer- 
so ence image and the non-reference image. 

Next, the operation of the third embodiment will be 
described. First, an image of an scene passes through 
the lens 101. the iris diaphragm 102. and the optical 
low-pass filter 103, and is formed on a photo sensing 
55 surfoce of the solid-state image sensor 1 04. At this time, 
the exposure controller 105 controls the iris diaphragm 
102 and the solid-state image sensor 104 so that a sin- 
gle scene is sensed a plurality of times in different expo- 
sures. 
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A plurality of images of the scene each of which is 
formed on the solid-state image sensor 104 in each 
image sensing operation are photo-electric converted 
and outputted as image data, and further analog-digital 
converted by the A/D converter 106. Then the digital 
image data of the plurality of images obtained as above 
is temporarily stored in the frame memories 107 and 
108. 

Exanrtples of images sensed by the sofid-state 
image sensor 104 are shown in Figs. 11 A and 11B. 
Figs. 1 1 A and 1 1 B are two images obtained by sensing 
an identical scene of the inside of a room having a win- 
dow in different exposures. 

Fig. 11A shows an image obtained by sensing the 
room so that the area B1 (inside of the room) is sensed 
in the proper exposure. In this image, the area A1 (out- 
side scenery through the wirxdow) is much brighter than 
the area B1 (inside of the room), and the dynamic range 
of the solid'State image sensor 104 is too narrow to 
properly express the both areas. As a result, the area 
81 (inside of the room) is sensed in proper exposure, 
whereas, the most brightest part of the area A1 (outside 
scenery through the window) is so bright that the corre- 
sponding part of the solid-state image sensor 1 04 is sat- 
urated and the parts are expressed in white. 

Fig. 1 1B is an Image obtained by sensing the iden- 
tical scene In a low exposure so that the scenery of the 
outside of the room is not expressed in white. In this 
image, the area A2 (outside scenery through the win- 
dow) Is sensed In proper exposure, however, the area 
B2 (inside of the room) is expressed darker than the 
corresponding area shown in Fig. 11B. 

Now, assuming that image data shown in Fig. 11 A 
is stored in the frame memory 107 and image data in 
Fig. 1 1 B Is stored in the frame memory 108, and letting 
the image data stored in the frame memory 107 be of a 
reference image, and the image data stored in the frame 
memory 108 be of a non-reference image. 

Conventionally, two whole images shown in Figs. 
1 1 A and 11 B are necessary for otnaining an image with 
a widened dynamic range. In contrast, in the third 
embodiment, the image data of the whole image shown 
In Fig. 11 A (reference Image) and image data of the 
area A2 In the image shown in Fig. 1 1 B (non-reference 
image) are needed, and Image data of the area B2 is 
unnecessary for the dynamic range expansion process- 
ing. 

In other words, by replacing the image data of the 
area A1 of the /oierence image shown in Fig. 1 1 B by the 
image data of the area A2 of the non-reference image 
shown in Fig. 1 1 B. an image expressing both the bright 
and dark areas in proper exposure can be obtained. As 
described above, the replacing area determination unit 
109 determines image data of an area which needs to 
be processed with the dynamic range expansion 
processing. 

Fig. 12 is a graph showing luminance levels of the 
reference image data shown in Fig. 1 1 A with respect to 
a horizontal line. e.g.. the line Q-Q*. As seen in Fig. 12. 
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in the area A1 which shows an outside scenery from the 
window brighter than the inside of the room, the solid- 
state image sensor 104 corresponding to the most 
brightest part (CI) is saturated. 

5 The determination unit 109 determines an area 
whose luminance level is over a predetermined thresh- 
old T so as to extract bright areas where the solid-state 
Image sensor 104 is saturated and the dynamic range 
expansion Is necessary In the reference image. Then. 

10 the area A2 in the non-reference image corresponding 
to the determined area Al in the reference image is 
determined as the area which needs to be processed 
with the dynamic range expansion processing. 

Next, the compression unit 110 picks up the image 

75 data of the area A2 determined by the determination 
unit 109 out of the entire image data of the non-refer- 
ence image, compresses the picked image data to gen- 
erate compressed data. Then, the storage unit 111 
stores the image data of the reference image and the 

20 compressed data as necessary data for the dynannic 
range expansion processing. At this time, information 
on exposure conditions (values of the iris diaphragm 
and shutter speed) used for sensing the reference 
image and the non-reference image, and information on 

25 the picked image data (e.g., a position of the picked-up 
area from the non-reference Image), are also stored 
with the compressed data. 

Note that as a method of picking up an area to be 
used in the dynamic range expansion processing from 

30 the non-reference Image, the following method can be 
used. As shown in Fig. 13. sensed Images are divided 
by n in the vertical direction and by m in the horizontal 
direction in advance, so as to obtain areas, w^i to Wpm- 
Then, the determination unit 109 finds an area, Wjj (1 s 

35 I ^ n. 1 ^ j ^ m). Including more than a predetermined 
number of pixels whose luminance levels are over a pre- 
determined value, and determines that the area is 
needed to be performed with the dynamic range expan- 
sion processing. 

40 In the aforesaid example, the image shown in Fig. 
1 1 A Is dealt with as the reference image and the image 
shown in Fig. 1 1 B Is dealt with as the non-reference 
image. However, it Is possible to use the image shown in 
Fig. 1 1B as the reference image and the image shown 

45 in Fig. 1 1 A as the non-reference image in the afbresakj 
dynamic range expansion processing. 

In such a case, the area B2. in the reference image 
shown in Fig. 1 1 B, whose luminance level is under a 
predetermined threshold is determined, then the image 

so data of the area B1 in the non-reference image shown In 
Fig. 1 1 A corresponding to the area B2 is picked up. and 
compressed image data of the picked image data of the 
area B1 Is generated. Thereafter, the compressed data 
of the area 61 arKi the Image data of the entire refer- 

55 ence inriage are stored as necessary data for the 
dynamic range expansion processing. 

According to the third embodiment as described 
above, it does not have to store entire image data of a 
plurality of sensed images, and image data of a refer- 
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ence image and compressed data, in a non-reference 
image, obtained by compressing picked image data of 
an area corresponding to an area, in the reference 
image, which needs to be processed with the dynamic 
range expansion processing. Therefore, it is possible to 5 
greatly reduce an amount of data to be stored as neces- 
sary data for the dynamic range expansion processing. 
In this case, since only a part of image data of the non- 
reference image is compressed and stored, an amount 
of data to be stored is reduced comparing to the case of 
compressing the entire image data of the non-reference 
image. 

(First Modification of the Third Embodiment) 

The first modification of the third embodiment will 
be described below with reference to Fig. 14. 

In Fig. 14, the same units and elements as those in 
Rg. 10 are referred by the same reference numerals, 
and explanations of those are omitted. In Fig. 14. refer- 
ence numeral 1 1 2 denotes a reference image data com- 
pression unit for compressing image data of a reference 
image. With the configuration shown in Fig. 14, instead 
of storing image data of a reference image, compressed 
reference image data generated by compressing the 
data of the reference image with the reference image 
data compression unit 112 is stored. Aocorcfingly. it is 
possible to further reduce an amount of data to be 
stored as necessary data for dynamic range expansion 
processing. 

Note that in the above<iesatbed third embodiment 
and its first modification, two images obtained by sens- 
ing a scene in different exposures are used to obtain a 
single image of a widened dynamic range, however, the 
present invention is not limited to this. For example, it is 
possible to apply the present invention to a case where 
more than two images are sensed in different expo- 
sures, i.e.. a case where a plurality of non-reference 
images are sensed. 

Further, by letting an image sensed in a proper 
exposure (standard image) be a reference image, when 
a single image with a widened dynamic range is pro- 
duced, since it does not have to reproduce the starxlard 
image from compressed data, it is easy to process data 
of a plurality of images to form a single image. 

Furthermore, in the third embodiment and its first 
modification, an area which is necessary for dynamic 
range expansion processing is picked up from a non- 
reference image and compressed data of the picked-up 
area is generated, then the compressed data and the 
image data of the reference image are stored as data 
necessary for the dynamic range expansion processing, 
however, the present invention is not limited to this. 

For example, the image data picked up from the 
non-reference image may be stored without being com- 
pressed, or image data of areas which need to be proc- 
essed with the dynamic range expansion processing in 
k)oth the reference and non-reference images may be 
picked up and stored after b^ng compressed or uncom- 



pressed. In either case, it is possible to reduce an 
amount of data to be stored for the dynamic range 
expansion processing than usual. 

(Second Modif ication of the Third Embodiment) 

Next, a second modification of the third embodi- 
ment will be described. In the second modification, dif- 
ferential data between a reference image and a non- 
reference image is stored as compressed data. Fig. 15 
is a block diagram illustrating a configuration of an 
image sensing app>aratus according to the second mod- 
ification of the third embodiment, and the same unit and 
elements as those in Fig. 10 are referred by the same 
reference numerals and explanations of those are omit- 
ted. 

In Fig. 15. reference numeral 113 denotes a lumi- 
nance level adjusting unit for adjusting the luminance 
levels of a reference image and a non-reference image. 
Further, reference numeral 1 14 denotes a differentiator 
for generating differential data for each pixel from image 
data stored in the frame memory 107 and image data 
stored In the frame memory 108. 

A part of an operation using the above-described 
image sensing apparatus, which is different from the 
operation described in the third embodiment will be 
explained below. Rrst assume that the frame memory 
107 stores image data of the reference image shown in 
Fig. 1 1 A. and the frame memory 108 stores image data 
of the non-reference image shown in Fig. 1 1 B. 

Under the above condition, the luminance level 
adjusting unit 113 adjusts luminance levels of the refer- 
ence image stored in the frame memory 107 and the 
non-reference image stored in the frame memory 108. 
Thereafter, the differentiator 114 generates the differen- 
tial data, i.e., compressed data, for each pixel from cor- 
responding pixels of the reference image and the non- 
reference image. 

For example, if the non-reference image is sensed 
in half of the exposure used to sense the reference 
image, and if the luminance level of the area B2 (inside 
of the room) in the non-reference image shown in Rg. 
1 1 B is half of the luminance level of the area B1 (inside 
of the room) in the reference image, then the luminance 
level of the image data of the non-reference image is 
doubled by the luminance level adjusting unit 113, then 
differential data for each pixel is generated by the differ- 
entiator 114. 

Then the image data of the reference image and 
the above compressed data as necessary data for the 
dynamic range expansion processing is stored in a not- 
shown storage medium by the storage unit 111. At this 
time, information on exposure conditions (values of the 
iris diaphragm and shutter speed) for sensing the refer- 
ence image and the non-reference image, and informa- 
tion on a condition for level adjusting for generating the 
differential data, and the like, are also stored with the 
compressed data. 
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According to the second modification of the third 
embodiment as described above, the image data of all 
the plurality of images necessary for the dynamic range 
expansion processing is not stored, but image data of a 
single reference image and compressed data gener- 5 
ated from the difference between the reference image 
and a non-reference image is stored, thereby greatly 
reducing the amount of data to be stored for the 
dynamic range expansion processing. 

Furthermore, according to the second modification 10 
of the third embodiment, differential data Is generated 
after luminance levels of a reference image and a non- 
reference image are adjusted, thus rt is possible to 
greatly increase data compression ratio comparing to a 
case of generating the differential data without adjusting is 
luminance levels of the reference image and the non- 
reference Image. 

The present invention can be applied to a system 
constituted by a plurality of devices, or to an apparatus 
comprising a single device. Furthermore the invention is so 
applicable also to a case where the object of the inven- 
tion is attained by supplying a program to a system or 
apparatus. 

The present invention Is not limited to the above 
enrttxxiiments and various changes and nrnxiifications 2S 
can be made within the spirit and scope of the present 
invention. Therefore to appraise the public of the scope 
of the present invention, the following claims are mada 

Claims 

1. An image sensing apparatus which senses an 
object a plurality of times in different exposures to 
obtain a plurality of images and makes a single 
image with a widened dynamic range by processing 3s 
the plurality of sensed images, characterized by 
comprising: 

histogram generating means (8) for generat- 
ing a histogram of signal levels of a first image 
sensed in a first exposure; 40 

deciding means (9) for deciding whether a 
dynamic range expansion processing is to be proc- 
essed or not on the basis of the histogram gener- 
ated by said histogram generating means (8); 

parameter determination means (10) for 45 
determining a parameter for sensing a second 
image of the object in a second exposure which Is 
different from the first exposure in accordance with 
a decision by sair! Jeciding means (9); and 

image processing means (1 2) for obtaining a so 
single image of the object by processing the first 
image and the second image which is sensed by 
using the parameter determined by said parameter 
determination means (10). 

55 

2. An image processing apparatus (49) which makes 
a single image with a widened dynamic range by 
processing a plurality of images obtained by sens- 
ing an object a plurality of times in different expo- 



sures, characterized by comprising: 

an interfece (42) for receiving the plurality of 
images; 

histogram generating means (45) for gener- 
ating a histogram of signal levels of a first image out 
of the plurality of received images which is sensed 
in a first exposure; 

deciding means (46) for deciding whether a 
dynamic range expansion processing Is to k>e proc- 
essed or not on the basis of the histogram gener- 
ated by said histogram generating means (45); 

parameter determination means (47) for 
determining a parameter for sensing a second 
image of the object in a second exposure which is 
different from the first exposure in accordance with 
a decision by said deciding means (46); and 

image processing means (48) for obtaining a 
single image of the object by processing the first 
inr\age and the second image which is sensed by 
using the parameter determined by said parameter 
determination means (47). 

3. The image processing apparatus according to 
claim 2, wherein said image processing apparatus 
(49) is used by connecting to an image sensing 
apparatus (41). 

4. The image sensing apparatus according to claim 1 
or 2, wherein said parameter determination means 
(10. 47) determines a plurality of parameters for 
sensing a plurality of images of the object in expo- 
sures which are different from the first exposure 
and from each other, and said image processing 
means (12. 48) processes the first image and the 
plurality of images sensed by using the determined 
parameters to obtain a single image of the object. 

5. The image sensing apparatus according to claim 1 
or 2. wherein, when said deciding means (9. 46) 
decides to perform the dynamic range expansion 
processing, said parameter determination means 
(10. 47) determines a parameter and said image 
processing means (12, 48) processes the first 
image and the second image. 

6. The image sensing apparatus according to claim 1 
or 2. wherein, when said deciding means (9. 46) 
decides not to perform the dynamic range expan- 
sion processing, said parameter determination 
means (10. 47) does not determines a parameter, 
the second image is not sensed, and said image 
processing means (12, 48) outputs the first image. 

7. The image sensing apparatus according to claim 1 
or 2. wherein said deciding means (9. 46) decides 
whether the dynamic range expansion is to be per- 
formed or not by comparing at least one total 
number of events occurred in at least one predeter- 
mined range of the signal levels in the histogram 
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generated by said histogram generating means (8. 
45) to a threshold predetermined by a range of the 
signal levels.. 

8. The image sensing apparatus according to claim 1 
or 2, wherein a parameter used for decision by said 
deciding means (9. 46) whether the dynamic range 
expansion processing is to be performed or not is 
set by a user. 

9. The image sensing apparatus according to claim 1 
or 2. wherein said parameter determination means 
(10, 47) determines a nuntber of images to be 
sensed in different exposures which are different 
from the first exposure and parameters for sensing 
the images on the basis of at least one total number 
of events in at least one predetermined range of the 
signal levels in the histogreun. 

10. The image sensing apparatus according to claim 9. 
wherein a user sets the nurr^ber of the images to be 
sensed in different exposures which is different 
from the first exposure and the parameters used for 
sensing the images instead of the parameter deter- 
mination means (10. 47). 

11. An image sensing method of sensing an object a 
plurality of times in different exposures to obtain a 
plurality of images and making a single image with 
a widened dynamic range by processing the plural- 
ity of sensed images, characterized by comprising: 

a histogram generating step of generating a 
histogram of signsd levels of a first image sensed in 
a first exposure; 

a deciding step of deciding whether a 
dynamic range expansion processing is to be proc- 
essed or not on the basis of the histogram gener- 
ated at said histogram generating step; 

a parameter determination step of determin- 
ing a parameter for sensing a second image of the 
object in a second exposure which is different from 
the first exposure in accordance with a decision at 
said deciding step; and 

image processing step of obtaining a single 
image of the object by processing the first image 
and the second image which is sensed by using the 
parameter determined at said parameter determi- 
nation step. 

12. The image sensing method according to claim 11, 
wherein, at said parameter determination step, a 
plurality of parameters for sensing a plurality of 
images of the object in exposures which are differ- 
ent from the first exposure and from each other are 
determined. arxJ at said image processing step, the 
first image and the plurality of images sensed by 
using the determined parameters are processed to 
obtain a single image of the object 



13. An in^ge sensing apparatus which senses an 
object a plurality of times in different exposures to 
obtain a plurality of images and makes a single 
image with a widened dynamic range by processing 

5 the plurality of sensed images, characterized by 
comprising: 

an image sensor (63) for sensing the object 
so as to obtain an image and outputting the 
obtained image; 

10 operation means (65) for finding a proper 

exposure by using a first image sensed in a first 
image sensing operation and a first exposure used 
to sense the first image; 

deciding means (65) for deciding whether a 

75 dynamic range expansion processing is to be per- 
formed or not; 

image sensor control means (67) for control- 
ling said image sensor (63) in order to sense a sec- 
ond image in an exposure which is different from 

20 the first exposure, on the basis of the proper expo- 
sure obtained by said operation means (65) and the 
first exposure; and 

image processing means (65) for, depending 
upon a decision by said deciding means (65), gen- 

25 erating a single image by using the first image and 
said second image. 

14. The image sensing apparatus according to daim 
13. wherein said image processing means (65) 

30 generates the single image without using the first 
image when a difference between the first exposure 
and the proper exposure obtained by said operation 
means (65) is over a predetermined value. 

35 15. The image sensing apparatus according to claim 
13, wherein the first image obtained in the first 
image sensing operation is off one frame. 

16. The image sensing apparatus according to daim 
40 13, wherein the first image obtained in the first 

Image sensing operation is of one field. 

17. The image sensing apparatus according to daim 
13, wherein, when the first exposure is the proper 

45 exposure obtained by said operation means (65), 
said image sensor control means (67) controls the 
image sensor (63) so as to sense the second image 
in the second exposure which is different from the 
first exposure and which is dedded on the basis of 

so luminance distribution of the first image. 

18. The image sensing apparatus according to daim 
13, wherein said image sensor control means (67) 
controls said image sensor (63) to sense a second 

55 image and a third image so that the first, second, 
and third images are sensed in proper, over, and 
under exposures on tiie basis of the proper expo- 
sure obtained by said operation means (65) and the 
first exposure. 
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19. The image sensing apparatus according to claim 
13. wherein said image sensor control means (67) 
controls said image sensor (63) so that it senses an 
Image by frame in an exposure which is different 
from the first exposure. 5 

20. The image sensing apparatus according to claim 
13. wherein said image sensor control means (67) 
controls said image sensor (63) so that it senses an 
image by field in an exposure which is different from io 
the first exposure. 

'21. An image sensing method of sensing an object a 
plurality of times in different exposures to obtain a 
plurality of images and making a single image with is 
a widened dynamic range by processing the plural- 
ity of sensed images, characterized by comprising: 

a first image sensing step (SI) of sensing a 
first image of the object and outputting the first 
image; so 

an operation step (S3) of finding a proper 
exposure by using the first image sensed at the first 
image sensing step (S1) and a first exposure used 
to sense the first image; 

a deciding step (S4) of deciding wrhether a ss 
dynamic range expansion processing is to be per- 
formed or not; 

a second image sensing step (S5. S7. S8. 
S10) of sensing a second image in a second expo- 
sure which is different from the first exposure, on 30 
the basis of the proper exposure obtained at said 
operation step (S3) and the first exposure; and 

an image processing step of, depending 
upon a decision at said deciding step (S4). generat- 
ing a single image by using the first image and the 35 
second Image. 

22. The image sensing method according to claim 21. 
wherein, at said image processing step, the single 
image is generated without using the first image 40 
when a difference between the first exposure and 
the proper exposure obtained at said operation step 
(S3} is over a predetermined value. 

23. The Image sensing method according to claim 21, 45 
wherein the first image obtained at the first image 
sensing step (S1) is of one frame. 

24. The image sensing mri^od according to claim 21. 
wherein the first image obtained at the first image so 
sensing step (SI) is of one field. 

25. The image sensing method according to claim 21. 
wherein, when the first exposure is the proper expo- 
sure obtained at said operation step (S3), at said ss 
second image sensing step (S5, S7. S8. S10). the 
second image is sensed in the second exposure 
which is different from the first exposure and which 



is decided on the basis of luminance distribution of 
the first image. 

26. The image sensing method according to claim 21 , 
wherein, at said second image sensing step (S5. 
S7. S8. SIO). a second image and a third image are 
sensed so that the first, second, and third images 
are sensed in proper, over, and under exposures on 
the basis of the proper exposure obtained at said 
operation step (S3) and the first exposure. 

27. The Image sensing method according to claim 21 . 
wherein, at said second image sensing step (S5. 
S7, S8, S10). the second image is served by frame 
in the second exposure which is different from the 
first exposure. 

28. The image sensing method according to claim 21. 
wherein, at said second image sensing step (S5. 
S7, S8. S10). the second image is sensed by field 
in the second exposure which is different from the 
first exposure 

29. An image sensing apparatus which senses an 
object a plurality of times in different exposures to 
obtain a plurality of images and makes a single 
image with a widened dynamic range by processing 
the plurality of sensed images, characterized by 
comprising: 

first compressing means (110) for generat- 
ing compressed data by compressing image data of 
an image which is other than a first image that is 
one of the plurality of images; and 

storage means (1 1 1) for storing image data 
of the first image arKi the compressed data of the 
image which is other than the first image. 

30. The image sensing apparatus according to claim 29 
further comprising: 

area determination means (109) for deter- 
mining an area which needs to be performed with a 
dynamic range expansion processing in the first 
image; and 

extraction means (110) for extracting an area 
corresponding to the area determined by said area 
determination means (109) in the image which is 
other than the first image. 

wherein said first compressing means (110) 
compresses image data of the area extracted by 
said extraction means (110). 

31. The image sensing apparatus according to claim 
29. wherein said first compressing means (110) 
includes level adjusting means (113) for adjusting 
luminance levels of the Image data of the first Image 
and the image data of the image which is other than 
the first image, and differential data generating 
means (114) for generating differential data 
k>etween the image data of the first image and the 
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image data of the image which is other than the first 
image. 

32. An image sensing apparatus which senses an 
object a plurality of times in different exposures to 
obtain a plurality of images and nnakes a single 
image with a widened dynamic range by processing 
the plurality of sensed innages. characterized by 
comprising: 

first compressing means (112) for generat- 
ing compressed data by compressing image data of 
a first image which Is one of the plurality of images; 

second compressing means (1 10) for gener- 
ating compressed data by compressing image data 
of an image which is other than the first image; and 

storage means (111) for storing the com- 
pressed data of the first image and of the image 
which is other than the first image. 

33. The image sensing apparatus according to claim 32 
further comprising: 

area determination means (109) for deter- 
mining an area which needs to be performed with a 
dynamic range expansion processing in the first 
image; and 

extraction means (1 10) for extracting an area 
con^esponding to the area determined by said area 
determination means (109) in the image which is 
other than the first image, 

wherein said second compressing means 
(1 10) compresses image data of the area extracted 
by said extraction means (110). 

34. The image sensing apparatus according to claim 
32. wherein said second compressing means (110) 
includes level adjusting means (113) for adjusting 
luminance levels of the image data of the first image 
and the image data of the image which is other than 
the first image, and differential data generating 
means (114) for generating differential data 
between the image data of the first image and the 
image data of the image which is other than the first 
image. 

35. An image sensing apparatus which senses an 
object a plurality of times in different exposures to 
obtain a plurality of images and makes a single 
image with a widened dynamic range by processing 
the plurality of sensed images, characterized by 
comprising: 

area determination means (109) for deter- 
mining an area which needs to be performed with a 
dynamic range expansion processing in a first 
image which is one of the plurality of images; 

extraction means (1 1 0) for extracting an area 
con-esponding to the area determined by sakJ area 
determination means (109) in the image which is 
other than the first image; and 

storage means (1 11) for storing image data 



of the first image and image data of the extracted 
area of the image which is other than the first 
image. 

5 36. An image sensing apparatus which senses an 
object a plurality of times in different exposures to 
obtain a plurality of images and makes a single 
image with a widened dynamic range by processing 
the plurality of sensed images, characterized by 

10 comprising: 

area determination means (109) for deter- 
mining an area which needs to be performed with a 
dynamic range expansion processing in a first 
image which is one of the plurality of images; 

15 extraction means (110) for extracting an area 

corresponding to the area determined by said area 
determination means (109) in the image which is 
other than the first image; and 

storage means (1 1 1) for storing image data 

so of the area, determined by said area determination 
means (109), of the first image and image data of 
the area, extracted by said extraction means (110), 
of the image which is other than the first image. 

25 37. The image sensing apparatus according to claim 
30. 33. 35 or 36. wherein said area determination 
means (109) finds an area represented by image 
data whose luminance level is greater than a prede- 
termined threshold or less than a predetermined 

30 threshold in the first image, and determines the 
found area in the first image as the area which 
needs to be performed with the dynamic range 
expansion processing. 

35 38. The image sensing apparatus according to claim 
30. 33. 35 or 36. wherein said area determination 
means (109) finds a predetermined sized area 
including a greater numl^er of pixels whose lumi- 
nance levels are greater a predetermined threshold 

40 or less than a predetermined threshold than a pre- 
determined value in the first image, and determines 
the found area in the first Image as the area which 
needs to be performed with the dynamic range 
expansion processing. 

45 

39. The image sensing apparatus according to claim 
29. 32, 35 or 36. wherein said first image is sensed 
in a proper exposure. 

50 40. An image processor having means for producing an 
image of expanded dynamic range from plural 
images sensed with different sensitivity or expo- 
sure, wherein image data of an image sensed for 
photometry is also used in produdng the image of 

55 expanded dynamic range. 

41 . An image processor having means for producing an 
image of expanded dynamic range from plural 
images sensed with different sensitivity or expo- 
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sure, wherein a first sensed image is used to deter- 
mine whether or not dynamic range expansion is 
desired, and is connbined or not with a second 
sensed image accordingly. 

5 

42. An image processor according to claim 41 wherein 
sensitivity or exposure for a second sensed image, 
is determined using the first sensed image. 

43. An image processor wherein plural images are io 
sensed with different sensitivities or exposures to 
obtain data for use in generating a single image of 
expanded dynamic range, wherein at least one of 

the plural images is stored in uncompressed form 
while at least part of one other is stored in com- is 
pressed form. 

44. An image processor wherein plural images are 
sensed with different sensitivities or exposures for 
use in generating a single image with expanded 20 
dynamic range, wherein parts of each sensed 
image required for expansion are identified prior to 
storage and storage performed only for those parts. 

45. An apparatus or method for image processing, or 25 
an electronic camera for still or motion pictures, 
having the features of any combination of the pre- 
ceding claims. 

30 
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FIG. 8 A 
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(54) Method and apparatus for image sensing with dynamic range expansion 



(57) An image sensing apparatus which senses an 
object a plurality of times in different exposures to obtain 
a plurality of images and makes a single image with a 
wk:lened dynamic range by processing the plurality of 
sensed images, generates a histogram of signal levels 
of an image sensed In a proper exposure by using a his- 
togram generator, automatically decides whether a 
dynamic range expansion processing is to be proc- 
essed or not on the basis of the generated histogram by 
using a dynamic range expansion deciding unit, deter- 
mines a parameter for sensing another image of the 
object in an exposure which is different from the proper 
exposure in accordance with a decision by the dynamic 
range expansion deciding unit by using a parameter 
determination unit, and obtains a single image of the 
object by processing the image sensed in the proper 
exposure and the image sensed by using the parameter 
determined by said parameter determination unit an 
image processing unit. When an image sensing appara- 
tus is not equipped with a photometer, a pre-sensing 
operation is performed, an image sensed in the pre- 
sensing operation is analyzed to find a proper exposure, 
and this Image Is also used in the dynamic range expan- 
sion processing. A part or whole image data of a plural- 
ity of images which are obtained as above and 
necessary for the dynamic range expansion processing 
are compressed by a compressing unit and the com- 
pressed Image data Is used in the dynamic range 
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